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In late 2015, NASA’s Aeronautics Research Mission Directorate (ARMD) funded an experiment in 
rapid design and rapid teaming to explore new approaches to solving challenging design problems in 
aeronautics in an effort to cultivate and foster innovation. This report summarizes several lessons 
learned from the rapid design portion of the study. This effort entailed learning and applying design 
thinking, a human-centered design approach, to complete the conceptual design for an open-ended 
design challenge within six months. 


The design challenge focused on creating a capability to advance experimental testing of autonomous 
aeronautics systems, an area of great interest to NASA, the US government as a whole, and an entire 
ecosystem of users and developers around the globe. A team of nine civil servant researchers from 
three of NASA’s aeronautics field centers with backgrounds in several disciplines was assembled and 
rapidly trained in design thinking under the guidance of the innovation and design firm IDEO. The 
design thinking process, while used extensively outside the aerospace industry, is less common and 
even counter to many practices within the aerospace industry. In this report, several contrasts 
between common aerospace research and development practices and design thinking are discussed, 
drawing upon the lessons learned from the NASA rapid design study. 


The lessons discussed included working towards a design solution without a set of detailed design 
requirements, which may not be practical or even feasible for management to ascertain for complex, 
challenging problems. This approach allowed for the possibility of redesigning the original problem 
statement to better meet the needs of the users. Another lesson learned was to approach problems 
holistically from the perspective of the needs of individuals that may be affected by advances in topic 
area instead of purely from a technological feasibility viewpoint. The interdisciplinary nature of the 
design team also provided valuable experience by allowing team members from different 
technological backgrounds to work side-by-side instead of dividing into smaller teams, as is 
frequently done in traditional multidisciplinary design. The team also learned how to work with 
qualitative data obtained primarily through the 70-plus interviews that were conducted over the 
course of this project, which was a sharp contrast to using quantitative data with regards to 
identifying, capturing, analyzing, storing, and recalling the data. When identifying potential 
interviewees who may have useful contributions to the design subject area, the team found great 
value in talking to non-traditional users and potential beneficiaries of autonomous aeronautics 
systems whose impact on the aeronautics autonomy ecosystem is growing swiftly. Finally, the team 
benefitted from using “sacrificial prototyping,” which is a method of rapidly prototyping draft 
concepts and ideas with the intent of enabling potential users to provide significant feedback early in 
the design process. This contrasts the more common approach of using expensive prototypes that 
focus on demonstrating technical feasibility. The unique design approach and lessons learned by the 
team throughout this process culminated in a final design concept that was quite different than what 
the team originally assumed would be the design concept initially. A summary of the more user- 
centered final design concept is also provided. 
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I. Introduction 


As a part of its strategy to foster innovation and transformation in aeronautics, NASA’s Aeronautics 
Research Mission Directorate (ARMD) portfolio includes efforts that promote higher risk studies and 
considerable flexibility in approach methods in a project named the Convergent Aeronautics Solutions 
(CAS) project. CAS focuses on short-term, high-risk efforts that converge multiple sources (from 
aerospace and non-aerospace) to design and rapidly assess transformative aeronautics concepts. This paper 
focuses on the lessons learned from a recently completed rapid, innovative design study in CAS dubbed the 
Aeronautics Autonomy Testbed Capability (AATC). 


The AATC study was launched in September 2015 with a two-pronged effort, one focused on technical 
deliverables and one focused on the teaming approach. The technical effort, which is the focus of this 
paper, focused on experimenting with rapidly learning and applying design thinking to complete a 
complex, conceptual design within 6 months. The teaming effort focused on experimenting with using a 
rapidly formed, purposefully ad hoc, team from multiple NASA centers to complete this task. While the 
lessons learned from the design thinking study form the foci of the current report, the influence of the 
teaming study are inseparable from some of the lessons learned and will be addressed where appropriate 
herein. A report on the rapid teaming study, conducted by organization scientists, is currently under 
development. 


As the AATC study was experimental in nature, learning and innovation were the highest goals. In this 
report, some of the most significant lessons learned are discussed with the hope to inspire and equip others 
in using non-traditional design approaches in aerospace. To frame the discussion of the lessons learned this 
report begins with a summary of the motivation for the fast-paced study of the AATC project, followed by 
a brief background on design thinking. The remainder of the report will be organized in lessons learned 
from using design thinking as compared to traditional engineering design approaches. A summary of the 
final conceptual design created by using design thinking is also summarized to provide an example output. 


II. Motivation 


As noted earlier, the NASA’s aeronautics research project CAS was created to “converge multiple sources 
to design and rapidly assess transformative aeronautics concepts.” CAS may be summarized as focusing on 
transformative concepts that are: 
1. Targeted on an ARMD Strategic Thrust 
2. Transformative and disruptive: project with high feasibility risk and are a substantial leap beyond 
the state of the art or state of practice as opposed to incremental advances 
3. Convergent: Integrating diverse knowledge to assess the feasibility of a transformative aeronautics 
concept. Include the convergence if multiple disciplines, competencies, and technology areas. 
Involve more than one NASA center, and use technologies from non-traditional partners from 
non-aerospace industries 
4. Feasibility Assessment Focused: Evaluate the best use of and the boundaries of the concept’s 
capabilities 
5. Rapid Execution: Completed in 6 months to 3 years with a bias toward research efforts that are 
less than 18 months in duration. 


The CAS project created the AATC effort to encapsulate these goals as well as to foster an innovative 
culture similar to a “design-shop.” These latter additional goals for the AATC project meant moving 
beyond simply conducting traditional aeronautics research faster with more disciplines. Embracing a 
“design-shop” approach meant embracing: working within a purposefully rapidly formed cross-discipline 
team that was geographically dispersed; moving considerably faster (6 months for an effort that would 
normally take 1.5 or more years); handling an ambiguous design challenge; experimenting with a highly 
unfamiliar approach (in this case design thinking); and adopting the non-traditional concepts that are 
inherent to the new approach (such as interviewing, handling qualitative data, sacrificial concepts). 


The AATC Team (and CAS leadership) recognized the need to propel NASA toward fast response to 
challenging assignments while embracing a rapidly changing aeronautics industrial landscape. This paper 
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highlights some of these challenges and successes. Differences with using design thinking are compared 
with conventional approaches that NASA uses for design and system development. 


Il. Background 


IIl.a. Brief Introduction to Design Thinking 

Numerous references describe the human-centered approach of design thinking with a few variations.'**° 
For the current study, the innovation and design company IDEO was contracted to work with the NASA 
team to use a design thinking approach. The design thinking approach commonly used at IDEO and 


Stanford are described herein? 


The iterative process used in the current study included the following key steps: 
e Start with an initial challenge or problem and a diverse team of designers 
Interview stakeholders, users, and beneficiaries to fully understand their needs 
Analyze and synthesize feedback from interviews and update the problem statement as needed 
Use rapid prototyping and sacrificial concepts to gather feedback from interviewees 
Refine concepts through feedback and iteration 


With the exception of the application of human-centered design approaches in human factors engineering, 
much of the aerospace engineering community infrequently applies design thinking. In some ways design 
thinking is counter to traditional aerospace research and development methodology. The lessons learned 
discussed subsequently will elucidate a few of these differences. 


Ill.b. The Technical Challenge 


Research on and the use of autonomous air vehicles continues to grow rapidly across the globe.°” A 
growing community of researchers and users is expanding far beyond the traditional aeronautics 
community. Historically dominated by defense needs, industries not traditionally involved in aeronautics 
with increasing interest in autonomous air vehicles include agriculture, film, real estate, sports, shipping, 
and hobbyists. These changes pose considerable challenges to regulators while opening the door to many 
new opportunities for users and raising diverse questions for researchers. 


Cognizant of these and other directions in the growth of autonomous air vehicles, NASA leaders in the 
ARMD posed a purposefully open question to the small, ad hoc, rapid design AATC team: 

“What experimental capability does NASA need to further autonomy research on unmanned 
aeronautical systems (UAS) such that a wide variety of autonomy research questions can be 
answered by many different researchers?” 

The AATC team was charged with completing the conceptual design phase of the new capability such that 
in later years NASA leadership can be better informed in determining whether or not to develop it. In the 
near term, the results can be used to inform research roadmaps. 


To allow the greatest opportunity for innovation, little additional requirements were leveled on the team. 
However, the following were offered to provide some guidance and clarification: 
1. Capability-Focused: Not specific or tailored to a reference mission or specific technology: 
a. Develop a user-friendly and adaptable capability: 
i. Capable of studying a wide variety of autonomy research topics (missions, 
technologies, concepts, etc.) 

ii. Useful to many different researchers, at different centers: Consider how this 
capability may be used by researchers from all four NASA aeronautics research 
centers (but the capability needn’t be physically located at each center) 

b. Several different UAS testbeds or vehicles, ground-based operations, etc., may be considered 
in creating the overall ARMD capability 
2. Think holistically to define the capability: Identify the entire “system” needed to implement the UAS 
test capability including the vehicles, the infrastructure, and the operational scenarios. 
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3. Include quantitative and qualitative analyses of using the testbed capability such as: cost, travel, time, 
ease of use, workforce training 


Design thinking was chosen as the design methodology for this effort for several reasons: to explore 
innovative approaches and due to the open-ended nature of the challenge question and socio-technical 
complexities in autonomy that include difficult technical challenges as well as the changing user 
communities and considerations related to economics, safety, regulations, and many other topics. As 
design thinking was new to most of the AATC team, the team quickly learned design thinking through 
facilitation and education from the contractor IDEO. The AATC team was composed of eight team 
members plus a principal investigator and was launched in the fall of 2015. 


IV. Lessons learned 


As noted earlier, six lessons learned are delineated herein. Each of these focus on topics where differences 
with conventional aerospace R&D methods are evident. The topics covered are: 

Beginning with No Detailed Requirements and Redefining the Problem 

Taking a Holistic Perspective Not Limited to Technical Feasibility 

Adopting an Interdisciplinary Versus Multidisciplinary Teaming Approach 

Tapping into the Power of Qualitative Data 

Discovering the Customer and Their Needs: Users, Stakeholders, and Beneficiaries 

Using Sacrificial Concepts Instead of Formalized Prototypes: 


me aos 


IV.a. Beginning with No Detailed Requirements and Redefining the Problem 


Conventionally, aerospace design and development begins with clear requirements that define, with 
specificity, important aspects of the desired system. The goal of well-defined requirements is to ensure that 
the customer’s needs are met. These requirements drive the entire technology development process 
providing the basis for decision making for the design and development team.*’There has been much 
discussion in recent years that challenge the conventional requirements-driven approach and offer other 
approaches such as a decision analysis framework.'*'' In this report, discussion will focus on contrasting a 
requirements-driven approach with a design thinking approach. 


The requirements-driven approach assumes that the customer fully understands their needs and can detail 
what is required to meet their needs. Thus, the problem statement is well formulated and a potential 
solution path has been identified. For highly complex and ambiguous problems, these assumptions may not 
hold. While the customer may have an identified need, the underlying problems related to the need are 
multi-faceted and, correspondingly, the potential solution paths are numerous. In these scenarios, the 
problem articulated by the customer must be explored so that a clearer understanding of the underlying 
factors and needs is obtained. Problem exploration effort often reveals underlying and unarticulated 
customer needs. At times, it becomes evident that the original customer problem was a symptom of a 
bigger problem. If the engineering design focused on what the customer originally articulated (a 
symptom), the larger problem (and the real need) would remain only partially addressed, if at all. For this 
reason and others, design thinking is inherently need-driven and not requirements-driven. 
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Figure 1. Problem redefinition. 


In design thinking, the initial phase of the process is often called “define” or “empathize” (with the user or 
customer) where the user’s problem is explored. In the context of this design study, this step entails 
understanding the points of view, contexts, and needs of users and other stakeholders. The problem is then 
iteratively redefined to ensure that the correct problem is being solved and the needs are sufficiently 
addressed. Figure | depicts the difference between a traditional requirements-driven approach and the 
design thinking approach. This iterative problem definition approach; while frustrating to some 
traditionally trained engineers who would prefer a fixed, known problem statement; also enables cognitive 
biases and ingrained assumptions of the design team to be challenged and updated. 


Numerous facets of the design thinking process facilitate the problem re-definition process. Five are 
discussed in this report: taking a holistic perspective; embracing diversity of thought through an 
interdisciplinary approach; using qualitative research techniques; having a user-centered approach; and 
using sacrificial concepts. Over-arching is the intrinsically iterative nature of the entire design thinking 
process where exploration and failure are embraced as necessary aspects of learning and improving the 
designer’s understanding of the problem to be solved. 


Recall that for the current study, the customer’s request was open-ended, complex, and ambiguous; lending 
itself to the design thinking approach. “What experimental capability does NASA need to further autonomy 
research on UAS such that a wide variety of autonomy research questions can be answered by many 
different researchers?” Many members of the design team, and innumerable others in the autonomy 
community within and outside of NASA, quickly assumed the solution would focus on designing one or 
more unmanned x-planes (small or large) and/or one or more simulators and/or labs. As the design 
thinking process unfolded, numerous underlying factors were identified that challenged our original 
assumptions. A few (of the many) findings that emerged during the problem re-definition effort were: 

1) There were countless dissimilar definitions of the word “autonomy” and attempts to create a single 


agreed upon definition were fruitless’ 

2) The UAS community was far more diverse than originally thought and diverged dramatically from 
traditional aerospace users to new industries with little to no aeronautics history (further 
discussion in the section titled “Discovering the Customer and Their Needs”’) 

3) Many UAS customers wanted to use their own vehicles 

4) The aerospace culture of high fidelity analysis and rigorous testing does not resonate with many 
new users who sought rapid demonstrations. 

5) The rapidly evolving autonomy ecosystem also requires NASA’s role in the evolving ecosystem to 
be re-considered. This was a significant challenge the team did not expect to have to address. 

These findings emerged from considerable effort to understand the latent needs of the aeronautics 
autonomy ecosystem. Several of the steps that were conducted to do so are discussed next.. 
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IV.b. Taking a Holistic Perspective Not Limited to Technical Feasibility 


Technical feasibility is the commencement and enduring focus of many engineering efforts at NASA, 
where questions such as “How can we enable this technical capability?” and “Will this operate as 
planned?” focus the efforts. In contrast, design thinking commences with user’s desires for the system and 
also considers the business viability of the system as well as the technical feasibility as shown in Figure 2. 
Questions such as “what does the user want” and “why do they want it” drive the early stages of the design 
effort. Traditional engineering assumes this information is sufficiently captured in the detailed 
requirements as noted above. 
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Figure 2. Holistic approach of design thinking. 


Adopting a combined people, business, and technical perspective facilitates the problem re-definition effort 
and reduces the tendency to have a “build it and they will come” perspective. Considering the desirability 
of a solution with respect to people involved in the problem area includes understanding users, 
stakeholders, and related industries. The business viability considerations include understanding what will 
be efficient, profitable, or otherwise needed to accomplish the mission of the industries involved. This may 
include the diverse considerations of non-profit entities such as the government and academia; the smaller 
resources of start-ups; and brand-recognition and the internal culture of existing companies. 


While providing a more balanced framework for stakeholder analysis, this holistic approach created some 
challenges for the traditional engineering design team. The most critical challenges of this approach were: 

1) Ensuring that the design team had or could obtain sufficient competence in non-engineering areas 
that include qualitative research methods traditionally used in the social sciences and business 
management. 

2) Synthesizing these diverse inputs into a realizable product. Integrating the varying qualitative and 
quantitative inputs that were sometimes contradictory required extensive iteration that was 
inevitability mixed with ambiguity 

3) Moving beyond working as a multidisciplinary team toward working effectively and efficiently as 
an interdisciplinary team. This lesson learned was significant and warrants a separate discussion 
in the next section. 

4) Working within a traditional engineering culture that was expecting a linear, step-wise approach. 
The holistic and inherently iterative approach of design thinking means progress is measured more 
in increased clarity and understanding of the problem and its potential solutions. While critical, 
these “products” were less tangible and measurable than traditional intermediary engineering 
deliverables that often include more measureable constructs such as early hardware components or 
versions of software. Managers may not understand or be able to effectively use traditional 
methods to evaluate, and thus reward, the quality or progress of this non-traditional approach. 
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For the current study, all of these challenges were present. Only one member of the design team had 
qualitative research methods training and none were formally trained in business management. Much 
needed business-related information was not available or was difficult to obtain. Synthesizing the diverse 
data obtained required an alternately divergent and convergent process that defied traditional methods of 
evaluating progress for both the design team and the organization in which we worked. Synthesis was also 
simply messier than working on a problem that was only technical. During the study, increased clarity and 
understanding of the problem domain was unmistakably obtained, but the design team lacked something 
tangible and quantifiable to show for this increased understanding. This latter challenge was mitigated by 
working within the CAS project that enabled a non-traditional approach to flourish. While CAS project 
management had no expectation of a linear, step-wise approach, the rest of the organizational culture still 
had this expectation, including members of the design team. 


IV.c. Adopting an Interdisciplinary Versus Multidisciplinary Teaming Approach 


Large and complex projects require expertise from multiple disciplines. Traditional multidisciplinary 
design practice often organizes project sub-teams according to their discipline; assigns each sub-team their 
respective part of the project; and specifies how these sub-teams are supposed to interface with each other. 
This structure often leads to conflict between the sub-teams, a diffusion of responsibility for tasks that may 
not exactly fit into a single discipline, and a lack of insights and new ideas that may be generated if the sub- 
teams were allowed to communicate with each other more freely. This organizational structure works well 
for simple design challenges or when incremental advances are needed. As technology progresses, there is 
a clear trend towards more complex and tightly integrated systems, so alternative teaming approaches 
should be pursued to better meet these modern challenges. 


Interdisciplinary teaming allows for more interaction between members with different skillsets and 
backgrounds. The idea is to bring people together throughout the entire design process instead of just the 
integration phase. A number of benefits can occur: 

1) False assumptions made by one group (e.g. power systems) about another group (e.g. thermal 
management) will be discovered and corrected early on in the design process when it’s much 
easier and cost effective to change the design. 

2) Technical challenges can be brought to light and examined by a more diverse group of people, 
which is a recipe for generating new insight. 

3) Members of the design team will be exposed to disciplines other than their own, creating 
opportunities for professional growth. 

4) Complex technical challenges that require integrated expertise from multiple disciplines will be 
more effectively addressed with an interdisciplinary team than a team with only partial or loosely 
connected knowledge of the challenge. 

5) The design process, as a whole, will be less contentious since everyone is essentially on the same 
team, embracing unity without requiring uniformity. 


Readers are referred to McGowan, et al’” to gain a structured understanding on various approaches to cross- 
disciplinary teaming: from traditional multi-disciplinary teaming with limited communication between 
teams to fully-integrated, collective work structures. Some approaches may be better suited for certain 
design challenges and organizations, so this gives management another “knob to turn” with respect to 
aligning their project structure to best meet the needs of each project. The design thinking process draws a 
design team to work in a more collective or interdisciplinary manner. For much of the AATC team, who 
were more familiar with working in a multidisciplinary approach where each researcher separately “owns” 
one aspect of system, adjustments to work style had to be made. In design thinking, the research, ideation, 
and synthesis steps all require the team to interact in an interdisciplinary manner where existing and new 
ideas are owned and iterated upon by the team. 


IV.d. Tapping into the Power of Qualitative Data 


The design research conducted in this study primarily used qualitative methods such as observation, 
ethnography, and interviewing, the latter of which formed the focus of the efforts. Though qualitative 
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research methods are used extensively in the social sciences and in design research, many in aerospace 
engineering are unfamiliar with these methods (with the exception of those who study human factors). To 
engineers trained in quantitative methods, qualitative methods can seem to contradict intuition. In a study 
of engineers who learned qualitative methodology, Borrego found that engineers who were used to the 
traditional engineering research methods struggled to understand a qualitative research approach.'* The 
NASA design team for this study, with strong backgrounds in quantitative research approaches, was no 
exception. 


The most significant lessons learned in using qualitative approaches on this study were: 
1) Existence: Acknowledging the non-numeric data obtained as real data; 
2) Analysis: Embracing the unfamiliar requisites of data synthesis that were both deeply 
interdisciplinary and time intensive; 
3) Maintenance: Learning how to handle the new “physical form” of the data (words, papers, and 
ideas, vs. numbers, codes, and equations); and 
4) Significance: Recognizing that some invaluable information could only be obtained qualitatively. 


Qualitative methods can provide a useful augmentation to quantitative studies to more richly illuminate 
processes, cultures, relationships, and motivations that impact a system’s design. As engineering design is 
a social, organizational, cultural, political, and a mechanical activity, only a diversity of research methods 
can help tap latent and unarticulated customer needs and enable improvements beyond current ideas and 
inherent biases.'* Engineering educators have long noted that training and application of qualitative 
research methods can improve engineering research, especially in design.'» '* '’ More training would have 
benefited the team in the current study. While the team quickly learned how to conduct effective 
interviews and do data synthesis, even after more than 70 interviews with diverse users, stakeholders, and 
beneficiaries, and obtaining both relevant and unexpected information, some team members asked: “how 
can we design something, we don’t have any data.” The desired data referenced in this quote were 
quantitative data. 


During the study, a profuse amount (more than could be used) of invaluable information was obtained 
using qualitative methods. The inductive and highly interactive approach to data analysis provided vital 
insights to the team on the user needs, stakeholder concerns, organizational constraints and opportunities, 
and potential technical solutions. The data synthesis and analysis process took extensive face-to-face time. 
More familiar with working in a divide-and-conquer style multidisciplinary approach, the interdisciplinary 
and constantly interactive nature of the data synthesis process required all team members to work 
differently as noted earlier. Team workshops required days of interactive discussion examples of the 
workshops are shown in Figure 3. 
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Figure 3. Team workshops and discussions. 


Readers familiar with the design thinking process recognize the use of posters and sticky notes used to 
capture the data synthesis, analysis and subsequent ideation. These tools were effective for several goals: 
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working quickly; working democratically; enabling quieter team members to share their thoughts and ideas; 
visualizing a very large number of ideas and notes simultaneously; and reorganizing, filtering, and updating 
analysis. However, the team was geographically dispersed and not able to keep the information gathered in 
one location for continuous and visual reference. Effectively maintaining the data on the posters and sticky 
notes long term was challenging. This study would have benefited from utilizing qualitative analysis 
software or other means to convert the posters and sticky notes into a format easier to manage from long 
distance. 


A significant benefit of the qualitative research methods used was found in obtaining information that could 
not be captured numerically. The examples of this are extensive, a few are noted below: 
1) Capturing the considerable differences in concerns and values of key stakeholders. 
2) Personally seeing the discomfort of potential users as they looked at early concepts and the 
excitement as they looked at final concepts. 
3) Hearing repeatedly that users did not want a traditional NASA x-plane. 
4) Appreciating the frustration in the autonomy communities inside and outside NASA with trying to 
identify and connect to needed skills, tools, etc. 
5) Embracing the professional diversity and rapid changes of the autonomy user communities. 


IV.e. Discovering the Customer and Their Needs: Users, Stakeholders, and Beneficiaries 


The very first part of our design process was to figure out who to talk to and what to ask them. Ina 
traditional process, we would have talked to our stakeholders and asked them for more details on what they 
wanted, and indeed, this was what our design team desired to do first. Engineers and researchers are taught 
to understand the needs of the customer (or funding official) so that we can better hit the target of what they 
are looking for. A major drawback to this approach is that the customer may not fully understand the 
solution space and even their own needs at a level that would be useful to designers. Furthermore, 
addressing only the needs of an individual or small group of stakeholders may not be the best overall 
solution to the design challenge. In large organizations such as NASA, it is unreasonable to expect high- 
level managers to be proficient in all of the technical developments related to their program space, thus it is 
just as unreasonable to expect them to provide prescribed solutions with detailed requirements for these 
types of complex design challenges. 


After accepting this premise, the design team initiated a brainstorming session to identify any possible 
group of individuals who may have an interest in the experimental evaluation of autonomous flight 
systems. The first groups on the list were no surprise: academic researchers, military, other government 
agencies, full-scale aircraft developers, and UAV developers. Purposefully looking outside of common 
groups such as these, we then broadened our thinking and began identifying other groups such as hobbyists, 
farmers, first responders, shipping companies, and community colleges. These types of users were 
categorized as more “extreme users” throughout this project, meaning that their experience with and 
relationship to autonomy is radically different from those of traditional user groups. Feedback from 
extreme users was especially beneficial to the design team because their needs were so unique and often 
underrepresented by existing technical solutions. For example, some of these users may be unfamiliar 
and/or uninterested in research, less familiar with airspace rules and regulations than a licensed pilot, and 
excluded from conversations between prominent technologists and key policy-makers. However, the needs 
of these users and the problems they are trying to address with UAS technology could potentially drive 
future regulatory and funding structures by means of creating entire new industries. 


Figure 4 depicts the interviewees ranked according to two factors; how much experience in autonomy they 
have and how “close” to NASA they were professionally. Autonomy experience was determined by the 
technical background of each interviewee, and their closeness to NASA was determined through the 
interviewing process and then ranking them on a scale of something like “never worked for or with 
NASA”, to “has contracted with NASA”, to “is currently employed by NASA at a field center’, up to 
“works at NASA HQ making high-level programmatic decisions.” Throughout the information gathering 
phase, this map was used to help ensure that the team is interviewing subjects from diverse backgrounds 
and not over relying on fellow employees that are conveniently available. This approach supports the 
holistic perspective used in design thinking as noted previously. 
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Figure 4. Interviewee user map. 


IV.f. Using Sacrificial Concepts Instead of Formalized Prototypes 


A key tenet of design thinking is rapid ideation, which was embodied in the form of sacrificial concepts 
during this project. Sacrificial concepts are ideas related to the problem at hand that are quickly developed 
into “prototypes” with the goal of enabling potential users, stakeholders, and beneficiaries (subsequently 
called “interviewees) to interact with and significantly modify early ideas. Sacrificial concepts are very 
rough early depictions of design concepts (a sketch, a cardboard mock-up, etc.) that helps the interviewee 
build on their understanding of their own needs and communicate these to the designer. Thus, instead of 
demonstrating near final technical functionality as is customary with most aerospace prototypes, sacrificial 
concepts are specifically designed to be changed by provoking reactions and prompting deeper 
consideration from interviewees to further discussion. 


The accuracy or even feasibility of the concepts are not important at this stage, as most will never survive 
beyond a description, a sketch, and perhaps a crude physical prototype. A good sacrificial concept draws 
from one or more key elements of a design challenge and offers creative solutions that may be considered 
far-fetched or even impractical; designers are given a reprieve from technical criticism here. Each 
addresses one or more user needs based on preliminary hypotheses about what the user may consider to be 
useful or desirable. These sacrificial concepts are then presented to interviewees with the intent that each 
sacrificial concept will elicit a reaction, positive or negative, followed by sparking discussion about how 
and why the interviewees feel the way they do about the concept as a whole or some aspect of it. This 
insightful discussion and the refined understanding of users’ needs, mental models, expectations, and 
perceptions that arises are the true products of using sacrificial concepts. 


The feedback obtained from discussing sacrificial concepts with interviewees is quite different than if they 
were simply asked direct questions about the design challenge such as what physical dimension to make 
something. One reason behind this is that people often allow personal biases to affect their answers to 
these questions, biases such as the desire to provide answers that might please the interviewer, holding back 
comments that may be controversial, fear of saying something “stupid”, or filtering feedback so that it falls 
in line with the interviewee’s employer’s positions. Another important benefit of using a sacrificial 

concept over a formalized prototype is that a greater understanding of the user can be obtained by watching 
them interact with the concept. This simple user-to-concept interaction can reveal latent needs, pull them 
out of their habitual thinking, and facilitate the user being more open, honest, and less constrained by the 
norms of their respective fields. 


During this NASA effort, the team developed over 20 sacrificial concepts that focused on many different 
elements of the autonomy design challenge. The team split into small groups and developed an initial set 
of concepts that reflected one or more key user needs identified from the initial round of interviews. A 
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description and an illustration were developed for each concept, an example of which is shown in Figure 5. 
User scenario storyboards were created to illustrate each concept in use, which helped the design team 
build upon and refine the first draft of each sacrificial concept. They were then presented to potential users 
and stakeholders. 
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Figure 5. Example of a team sacrificial concept. 


During the next team workshop, the feedback gathered was presented to all members of the design team 
and synthesized into a refined set of user needs that were much more insightful than those obtained through 
direct questioning of the interviewees. These needs were then used to produce both new concepts and 
refinements to the existing ones, driving an evolution towards increasingly useful and practical solutions. 
As the design team worked in an interdisciplinary manner, elements from several different sacrificial 
concepts were brought together to drive towards the final design solution. 


The iterative process of feedback and refinement gradually advances the sacrificial concepts to what can be 
considered preliminary design concepts and then to final design concepts that resemble more traditional 
engineering design concepts. By moving through this progression, the needs of the users are more likely to 
be addressed and less resources are spent on concepts that insufficiently address user needs. Also during 
this progression, technical considerations and feasibility play an increasing role in the development and 
refinement. Additionally, several technically feasible preliminary design concepts that were insufficient for 
the current user group but useful for others were also created and ideas from these concepts were later 
incorporated into other NASA projects at the conclusion of the current design challenge. 


V. How Lessons Learned Impact Final Concept Framework 


The lessons learned described above enabled several design concepts to be created that would not have 
arisen from using a traditional approach. To demonstrate this, a summary of the final conceptual design is 
provided below. Readers are reminded that this conceptual design output is the preliminary conceptual 
design output from the AATC rapid innovation experiment. While insights from this conceptual design 
have influenced further research, the design concept described does not represent an official position of 
NASA. As such, no NASA funding decisions have been made regarding this concept. The conceptual 
design concept summary is presented here to provide an example of the output of applying design thinking 
to a complex aerospace challenge. Those interested in more details of the design concept are requested to 
contact the authors who can provide additional information and connections at NASA. 


The final conceptual design derived by the team is presented below beginning with a description of the 
users and the user needs addressed. The scope of considerations for this design effort was enormous and 
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narrowing to a single design direction proved to be a great challenge. Innumerable facets were considered 
in developing the final design concept and only a small portion of these facets are illuminated here. Several 
other design concepts were also considered by the team and elements of these concepts have been applied 
to other research efforts. Over-arching however, is the impact applying the design thinking process has had 
on each individual team member. Since the time of this study, most team members have recognized a 
change in the way they approach different portions of their new projects where they apply one or more 
aspects of design thinking. 


V.a. The User 


It was not possible for the team to synthesize a design concept that would satisfy all autonomy users, or 
even all the users that were interviewed. A smaller and manageable subset of the users was needed to make 
forward progress. To that end, after considerable analysis in understanding the UAS user ecosystem, the 
team selected a group of autonomy researchers (users) that need to operate multiple vehicles with a reduced 
crew beyond visual line of sight (Fig. 6). These users were identified as UAS developers, payload 
developers, flight operations crew members, and multi-vehicle system developers. The users were selected 
because synthesis of the design research identified them as having the largest impact, being the newest field 
of research, being a most difficult problem, and possessing a need to stay ahead of the technology ‘curve’ 
to be competitive. 


Figure 6. Safe, routine flight of multiple vehicles, beyond visual line of sight, with reduced crew 


Design research revealed seven user needs to be addressed: 

1) Beyond visual line of sight solution, 

2) Reduced crew, 

3) Scalability, 

4) Integration challenges, 

5) Need for a connected community to access a multitude of technical resources for operations, 

6) Need for vehicle and personnel safety, and 

7) Test site selection challenges. 
These seven needs (Fig. 7) were common for researchers studying both vehicle autonomy and operations 
autonomy, and provided a framework to serve the middle ground for both communities. 
Based on these users, a concept direction (framework) was created. More traditionally, final designs are 
tangible objects such as a vehicle, a lab, a code, etc. However, one unusual output from the design thinking 
process was that the design team recognized that the identified users need an integrated and multi-pronged 
solution, some of which was not tangible, but nevertheless necessary. The design team also recognized that 
a progressive solution was necessary — one that is designed to continually respond to the changes in 
autonomy. Given the pace of change in the autonomy ecosystem, more commonplace “static” solutions, 
such as a suite of vehicles, would not address some underlying needs nor the rapid changes in needs 
anticipated in the near future. The framework represents a strategic approach to a design concept where the 
details of how to implement each part of the framework are not yet completed and need considerably more 
effort to finalize. 
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Vehicle Autonomy Operations Autonomy 


The needs and associated barriers identified in this user group 
encompass the majority of the autonomy research users. 


Figure 7. User needs. 


The framework (Fig. 8) of the final design concept was divided into three main parts to address the most 
significant needs: 

A. The need for a connected community (or ecosystem) of autonomy researchers and users, 

B. The need for new flight processes (within NASA) and 

C. The need for uniquely designed and integrated testbeds, both new and existing. 
The framework was intended to connect the three parts so they build-on and support one another such that a 
better connected community of UAS users would provide the foundation for designing the new flight 
processes, which would then provide the foundation for designing new testbeds, which, cyclically, would 
reward and further connect the community and produce new processes and so on. Ultimately, the final goal 
is a nexus of all three parts of the framework. 
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Figure 8. The concept framework 
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V.b. Connecting the Community 


This portion of the framework was intended to establish connections to current communities of practice that 
are engaged in the identification, assessment, development, advancement, and exploitation of technologies 
that enable autonomous systems. From Fig. 8, some examples of the community include NASA 
aeronautics research centers, technical developers of autonomy technologies, persons supporting flight 
operations, and government regulators (such as the FAA). From user interviews, these groups all suffer 
from working in a technology area that is too big to know and have related communication, cultural, and 
organizational barriers. Users expressed needs to be connected to other organizations doing similar work. 
Specifically, NASA researchers had needs to leverage expertise and get connected to (internal and external) 
groups at the forefront of autonomy research. User interviews uncovered a strong desire for NASA to stay 
relevant in the community. NASA was found to have strong value in its ability to disseminate knowledge, 
leverage community expertise, lead, and enable “small-but-important” (SBI) contributors (such as a small 
business or individual researcher). Some key features were uncovered that need to be addressed to make 
this part of the framework successful. They included: providing incentives for collaboration; having 
trained collaboration professionals; removing ‘soft’ barriers to communication; and encouraging 
interdisciplinary work. 


Two prototype ideas were developed by the team to encourage connections in the community, Fig 9. The 
first was conceptually labeled as an Autonomy Accelerator Team (AAT). The AAT was a “round-table” 
concept that would establish new roles within NASA to specifically be autonomy research connectors. 
These well-trained connectors would utilize tools like the design thinking process and other user-centric 
approaches. These personnel would need technical engineering knowledge as well as social and 
communication skills. If successful, the AAT would unify researchers, incentivize collaboration, have an 
agency-wide impact, identify research and development challenges, and facilitate innovation. The second 
concept was labeled NASA’s Autonomy Portal. This portal was envisioned as a mobile and web-based 
tool to help “connect-the-dots” (researchers and research) and to pool and organize autonomy related 
information within NASA to facilitate more efficiently and effectively accessing the diversity of research in 
the autonomy research landscape. 


Recalling the AATC lessons learned, one of the key lessons was the ability to redefine the problem and 
arrive at a better overall solution while using qualitative measures to listen to users. This portion of the 
framework served as the key example of redefining the problem and using qualitative data: the design team 
was able to identify an oft-repeated user need to establish a better connected autonomy community. 
Traditionally, the design team would not have considered a social-based solution and would have designed 
a specific piece test hardware. In the early stages of the AATC study, sacrificial concepts of test hardware 
were created and users consistently noted this was not their biggest need. Given NASA’s history of 
working on a requirements-based approach that typically focuses on technical details, the community- 
focused solution would have been well outside normal project boundaries. 
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Figure 9. Community connection prototypes: 
(a) Autonomy Accelerator Team (combine technical and social team skills) and 
(b) Autonomy Portal 
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V.c. Flight Processes 


This portion of the framework was intended to update NASA flight operations to a state needed by 
autonomy research teams based on existing technical knowledge and best practices. The new flight 
operations would be facilitated by connecting and continuously learning from flight operations groups 
external and internal to NASA. From Fig. 8, the driving need is to accommodate autonomous flight 
beyond visual line of sight (BVLOS) with multiple vehicles while using a reduced flight crew. 


From interviews, users experience barriers to accelerated progress in areas that include NASA flight and 
safety processes, communications, and organizational cultural differences. Reducing barriers in these areas 
would accommodate user needs to perform routine and safe research with multiple vehicles in flight with a 
reduced crew at an accelerated pace (compared to current NASA research pace) and at a reseasonable cost. 
NASA was found to have a strong leadership role in aviation research, but has no current capabilities to 
perform research as listed above. Significant collateral benefits can be enabled by creating new flight 
processes that support routine and safe autonomous research as described above. Key features of better 
flight processes include (a) improved inter-connection of NASA flight operations groups at different 
centers, (b) facilitating new test capabilities, (c) working with users to drive operational standards, and (d) 
development of new flight CONOPS and procedures (Fig. 10). 
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Figure 10. Conceptual Design of New Flight Operations 


One prototype is included in this report as an example of how to develop new flight processes. The 
prototype would look like a NASA special operations team made up of NASA flight process stakeholders 
from all NASA aeronautics centers paired up with dedicated design thinking practitioners. Team personnel 
would have a full-time commitment to conduct design research, synthesize, and produce new NASA flight 
operation and safety processes. The special operations team would provide rapid collaborative process 
development, human-centric design, and have a leading role in the community as they remain connected to 
the latest and most advanced flight research in autonomy. 


From the lessons learned from the AATC study, interdisciplinary teaming and discovering the user allowed 
the entire AATC team to understand that many, or all, aspects of autonomy research need improvements in 
flight and safety processes. Even though many interviewees talked about how flight process improvements 
would allow faster progress than would normally occur, the multi-faceted flight process needs were not 
obvious to all team members. The design team learned that this aspect of the final concept needed a balance 
between NASA’s strength of safety and rigor and the strength of some external users who moved more 
quickly and were more innovative. The AATC team also realized that the user communities are constantly 
and rapidly evolving such that a one-time flight process re-design would be quickly antiquated. This part of 
the strategic design framework embraced continuous design as a key feature of keeping flight processes 
instep with and potentially ahead of the fast-paced autonomy ecosystem. The interdisciplinary nature of 
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the AATC design team and the interactions with diverse user groups made it possible to understand and 
communicate these needs to all team members and obtain consensus. 


V.d. Testbeds 


This element of the framework (Fig. 11) would facilitate and develop Unmanned System Testbeds (UST) 
where the focus is on the integrated design of autonomous vehicles with autonomous operations and 
technologies. From design research, it became clear that many users were focused on one aspect of 
autonomy (vehicle or operations or technology). And, many user’s goals were hampered by trying to 
retrofit the wrong operations with the wrong vehicle with ill-fitting technologies for example. In interviews, 
users experienced barriers to accelerated progress that include access to flight testing infrastructure and 
technologies as well as access to relevant skills in the autonomy ecosystem (addressed with the community 
part of the design) and needing updated and sometimes unique flight procedures (addressed with the 
processes part of the design). Users also have a number of testing needs that include (1) access to research 
rich test environments, (2) access to expertise in system integration, and (3) use of their own test vehicles 
rather than being constrained to an existing test article. Through the design research process, NASA was 
found to have a strong leadership role in many areas such as being a highly trusted organization and having 
a desire to support innovation, understanding risk, and possessing test experience. Key features of the 
testbed framework include the ability to design and build unique and safe testbeds while simultaneously 
integrating the vehicles, operations, and technologies. This also facilitates improving hardware and 
software standards. The integrated system design approach would provide needed feedback to drive 
improvements in autonomous vehicle technology and flight operations. 


Sean 
aed 


Vehicles 
f™ 
SQN UST 
ro 
Operations Technologies 
el 


Figure 11. Unmanned System Testbeds (UST) 


One example prototype is the mobile autonomy testbed. This testbed could be a UST with the ability to 
safely transport test equipment and aircraft to data rich environments. It would then allow and enable flight 
operations with a greatly reduced crew in the data-rich environment. This would enable many aircraft to 
fly and collect autonomy research data in a variety of relevant environments (urban, forest, water, etc...). 
The capability would be built by working with the more connected autonomy community and would have 
newly designed operational capability for multiple vehicles safely controlled by one operator. 


Referring back to the project lessons learned, the testbed portion of the framework was heavily influenced 
by the use of sacrificial concepts. At one point in the design process, many sacrificial concepts contained 
the use of team-proposed facilities and autonomous test vehicles. While some users had favorable 
responses to these early concepts, surprisingly, many users had strong negative responses to these 
sacrificial concepts. In fact, the team learned that users wanted access to relevant flight environments but 
using their own hardware, software and vehicles — an approach that would not have been obvious apriori. 
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VI. Concluding Remarks 


While aeronautics has long inspired innovation in many areas, large organizations such as NASA are 
mindful that continuing to pursue innovation requires taking purposeful steps to support the exploration of 
non-traditional approaches. Toward that end, NASA’s Aeronautics Research Mission Directorate launched 
a rapid design and rapid teaming experiment in the fall of 2015 to explore new approaches to rapidly 
responding to challenging problems. This effort, called the Aeronautics Autonomy Testbed Capability 
(AATC) study, explored the use of design thinking to address an open-ended autonomy challenge with a 
rapidly formed, multi-center team. During this study, design thinking was learned and applied to complete 
the conceptual design of an autonomous experimental capability in a 6 month period. 


This paper distilled the key lessons learned from using the human-centered design thinking approach by 
contrasting this approach with more common approaches used in aerospace research and development. 
While design thinking is not appropriate for all problems, several aspects of the process can inspire 
significant innovation. The AATC experiment was transformative for the team members involved due to 
the unique design approach used and the teaming approach used. Based upon the lessons learned, we close 
with a few final insights for future design teams: 
1. For more complex problems, the customer or user will not be able to provide all requirements and 
all information. 
2. Non-technical aspects of the problem space may be significant and need to be addressed in the 
design. 
3. Some of the best data may not be captured in numbers. 
4. Learn how to use qualitative research methods and continuously learn about the users, the 
stakeholders, and the beneficiaries of the system and their context. 
5. Simple, draft sacrificial prototypes can yield considerable insight for the users and stakeholders 
and will help the design team communicate. 
6. Challenge your assumptions about the design concept by welcoming early feedback. 
Embrace diversity of thought instead of necessitating uniformity. 
8. Step-wise and predictable linear processes are not always better. 
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